Skip to content

feat(core): support DataBlock in all provider message converters#1933

Open
Buktal wants to merge 2 commits into
agentscope-ai:mainfrom
Buktal:feat/data-block-converter-support
Open

feat(core): support DataBlock in all provider message converters#1933
Buktal wants to merge 2 commits into
agentscope-ai:mainfrom
Buktal:feat/data-block-converter-support

Conversation

@Buktal

@Buktal Buktal commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

AgentScope-Java Version

1.0.12 (development)

Description

DataBlock is documented as the forward-looking unified container for all binary modalities (image / audio / video / file), replacing the legacy ImageBlock / AudioBlock / VideoBlock subclasses. However, none of the provider message converters handled it — when a user built a message with DataBlock, every provider silently dropped the block and the LLM received no media content.

This PR fixes that gap.

Changes:

  • URLSource: Add optional mimeType field for extension-less URLs (CDN signed URLs, API-generated media links). Backward-compatible — existing callers using the single-arg constructor or omitting mime_type in JSON are unaffected. MIME type resolution priority: Base64Source.mediaType > URLSource.mimeType > extension-based inference via MediaUtils.determineMediaType.

  • DashScope: Add convertDataBlockToContentPart to DashScopeMediaConverter; add DataBlock branch to DashScopeMessageConverter (both convertToMultimodalContent and hasMediaContent / convertContentBlocks).

  • Gemini: Add convertToInlineDataPart(DataBlock) to GeminiMediaConverter; add DataBlock branch in GeminiMessageConverter; fix extractSourceFromBlock which previously threw on DataBlock.

  • Anthropic: Add convertDataBlock to AnthropicMediaConverter (image-only — Anthropic SDK does not expose a generic binary content block param); add DataBlock branch to AnthropicMessageConverter.

  • OpenAI: Add DataBlock branch to OpenAIMessageConverter (routes to image / audio / video based on MIME prefix); add resolveMimeType utility to OpenAIConverterUtils.

How to test:

Msg msg = Msg.builder()
    .role(MsgRole.USER)
    .content(DataBlock.builder()
        .source(URLSource.builder()
            .url("https://example.com/photo.jpg")
            .build())
        .build())
    .content(TextBlock.builder().text("Describe this image").build())
    .build();

Previously the image was silently dropped; now it is routed correctly to each provider's image slot.

For extension-less URLs:

URLSource.builder()
    .url("https://cdn.example.com/media/abc123")
    .mimeType("video/mp4")
    .build()

Closes #1932

Checklist

  • Code has been formatted with mvn spotless:apply
  • All tests are passing (mvn test)
  • Javadoc comments are complete and follow project conventions
  • Related documentation has been updated (e.g. links, examples, etc.)
  • Code is ready for review

- Add optional mimeType field to URLSource for extension-less URLs
- Add convertDataBlockToContentPart to DashScopeMediaConverter
- Add DataBlock branch to DashScopeMessageConverter (multimodal path and hasMediaContent)
- Add convertToInlineDataPart(DataBlock) to GeminiMediaConverter
- Add DataBlock branch to GeminiMessageConverter; fix extractSourceFromBlock
- Add convertDataBlock to AnthropicMediaConverter (image modality)
- Add DataBlock branch to AnthropicMessageConverter
- Add DataBlock branch to OpenAIMessageConverter (image/audio/video routing)
- Add resolveMimeType utility to OpenAIConverterUtils

Closes agentscope-ai#1932
@Buktal Buktal requested a review from a team June 26, 2026 07:38
- Remove validateImageExtension in AnthropicMediaConverter.convertDataBlock:
  mimeType is already verified as image/* before the URL branch, so the
  extension check would break extension-less CDN URLs with a mimeType hint
- Replace lastIndexOf-based extension extraction in GeminiMediaConverter with
  MediaUtils.getExtension to correctly handle signed URLs with query-string dots
- Remove dead validateExtension parameter from DashScopeMediaConverter.sourceToUrl
- Add DataBlock branch to AbstractBaseFormatter (hasMediaContent,
  convertToolResultToString, getSourceFromBlock) — affects all providers
- Add DataBlock branch to GeminiConversationMerger (multi-agent history merge)
- Add DataBlock unit tests to DashScopeMediaConverterTest,
  AnthropicMediaConverterTest, and GeminiMediaConverterTest
@itxaiohanglover

Copy link
Copy Markdown
Contributor

Good addition! Supporting DataBlock in all provider formatters ensures consistent handling across providers. The hasMediaContent check and convertToolResultToString updates are thorough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support DataBlock in all provider message converters

2 participants